Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: Administrator, Curator, or Staffbleblanc Date: Jun 18, 2003 3:08am
Forum: freecache Subject: Questions regarding Freecache

(I originally posted this on the Audio board, but figured this would be a better forum)

Freecache looks really cool, but I've got some questions after reading about it.

1. Say for example that I want to download file A from the Archive and I'm the first person to use Freecache to do this. If I understand correctly, since I'm the first person to request it via Freecache it won't be found on another server yet, is that right?

2. While I'm downloading file A (still as the first person to do so), the file is also being copied to a FreeCache server at the same time for other folks. Is that correct? So essentially I would be downloading the file from the Archive through a FreeCache server linked to it, and when my copy is complete then the FC mirror would also have a complete copy too.

3. Does Freecache load balance? In other words, can Freecache determine which mirror will provide me as a downloader with the best speed (if there's more than one)? Can it also determine if the Archive itself will give me the best speed and just go back to the source instead? Or, will it just give me the closest mirror?

4. Since it's all HTTP (and the files I'm interested in are tarred together as one enormous file), resuming will probably be an issue. Can a download client be used with Freecache?

5. What about the IAFM? Can that be integrated with Freecache? If not, which would you prefer that us downloaders use (which of the 2 technologies is more promising)?

6. Is Freecache able to determine file integrity before hosting the file at a remote site?

7. I see mention of ISP's here, are there any that are working with us or are the other servers just volunteers with a server on a big pipe?

Just to drop some info, I'm testing FC by downloading SCI 1/31/2000 at an average speed of 9.71KB/sec. I must be the first. :)

Mainly I'm just trying to understand more how it works and treats the ordinary download. Thanks for reading.

-Brad

This post was modified by bleblanc57 on 2003-06-18 10:08:25

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffbrewster Date: Jun 18, 2003 4:49am
Forum: freecache Subject: Re: Questions regarding Freecache


>>Good questions. I have interwoven answers with ">>"

(I originally posted this on the Audio board, but figured this would be a better forum)

Freecache looks really cool, but I've got some questions after reading about it.

1. Say for example that I want to download file A from the Archive and I'm the first person to use Freecache to do this. If I understand correctly, since I'm the first person to request it via Freecache it won't be found on another server yet, is that right?

>>Correct.

2. While I'm downloading file A (still as the first person to do so), the file is also being copied to a FreeCache server at the same time for other folks. Is that correct? So essentially I would be downloading the file from the Archive through a FreeCache server linked to it, and when my copy is complete then the FC mirror would also have a complete copy too.

>> Not quite. If it is not in a FreeCache yet, then you get redirected to the original. When enough people request it (currently over 2) then a FreeCache will be told to get a copy of the original. Subsequent requests will go to that FreeCache or other FreeCaches that have a copy.

3. Does Freecache load balance? In other words, can Freecache determine which mirror will provide me as a downloader with the best speed (if there's more than one)? Can it also determine if the Archive itself will give me the best speed and just go back to the source instead? Or, will it just give me the closest mirror?

>>Ralf is trying different policies. The current one is to go to the FreeCache that has the file. If there are multiple, then it tries to find the closest.

4. Since it's all HTTP (and the files I'm interested in are tarred together as one enormous file), resuming will probably be an issue. Can a download client be used with Freecache?

>> the current script does not handle ranges unless apache is magically doing it. This will come later as need arises. Please try the download clients and let us know if it works.

5. What about the IAFM? Can that be integrated with Freecache? If not, which would you prefer that us downloaders use (which of the 2 technologies is more promising)?

>> Please use FreeCache. We would like to integrate IAFM, but this may take awhile. IAFM is nifty for bringing down a bundle (freecache does this with a tar file), restarting (freecache does not handle this yet), and bandwidth sharing (but it only gets 12% from peers).

>>we are shooting for 10x more bandwidth from others or 1000%, so IAFM has a long way to go. Freecache has a good shot at getting there quickly.

6. Is Freecache able to determine file integrity before hosting the file at a remote site?

>>yes, it uses md5 and seems to work robustly.

7. I see mention of ISP's here, are there any that are working with us or are the other servers just volunteers with a server on a big pipe?

>>we are currently working with researchers and those wanting to build the commons: read Universities. when it stabilizes, we would like to expand.

Just to drop some info, I'm testing FC by downloading SCI 1/31/2000 at an average speed of 9.71KB/sec. I must be the first. :)

>>Yuck. You should be able to see in the URL when it redirects you to a freecache. this depends on the browser, unfortunately.

>>to find a file that will be fast, try going to the status2 page (you go to www.freecache.org and then go to status, then click on any of the freecache links: this will show you what files are in each freecache. Please try them out and give feedback-- we have seen speeds up to 30Mb/sec.

Mainly I'm just trying to understand more how it works and treats the ordinary download. Thanks for reading.

>>thank you for helping. as this system stabilizes, we would like everyone to be using freecache rather than our direct servers. this way, we can scale up and still deliver great stuff fast.

>> -brewster

-Brad

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Jun 19, 2003 11:09pm
Forum: freecache Subject: Re: sliding an item into a freecache

Great Q&A, thanks, it really helped me understand enough to dip a toe in!

If it is not in a FreeCache yet, then you get redirected to the original. When enough people request it (currently over 2) then a FreeCache will be told to get a copy of the original. Subsequent requests will go to that FreeCache or other FreeCaches that have a copy.

So, while I was playing around with an item that was not yet in any cache as of last night, I started d/l (over a typical home cable modem, not I2) from the Freecache link once or twice and pulled ~16 K/s- until I stopped due to thunderstorm.

This morning I restarted on the same item with the same connection, and was getting it at 198 K/s (until I stopped again due to another thunderstorm, dangit). It must have gone into a cache O/N based on my prior requests.

So, did I just stumble onto a workaround for getting meaty downloads on lower-demand items- "tease" Freecache with a couple of brief d/l tries to force the item into a cache before hooking into a full, fast download?

This post was modified by hamilton on 2003-06-20 06:09:27

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffJonathan Aizen Date: Jun 20, 2003 1:17am
Forum: freecache Subject: Re: sliding an item into a freecache

My understanding was that FreeCache is supposed to detect when a user cancels. What it does when it detects that, though, I'm not sure. Ralf or Brewster would know best.

I thought of this myself the other day. It's an interesting policy question.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffRalf Muehlen Date: Jun 20, 2003 4:10am
Forum: freecache Subject: Re: sliding an item into a freecache

Your download speed depends to which FreeCache you get redirected to. For the current beta testing that's random; in the future it'll be based on your IP#. Some FreeCaches are faster than others, but it also depends on how much other traffic is going on. So your download speeds WILL vary.

I'm in the middle of revamping the policies of when and where a file gets cached. More popular items will get cached before less popular items.

And yes, there are ways to trick FreeCache into caching a file. For now, I treat all download attempts the same. In case this becomes a problem, I can distinguish between valid requests and "trick" request, and discard the latter. I'd rather not do that though.

Think of FreeCache as a commons. You can take more than your fair share, but it'll cost the other users. In other words, a file with false popularity takes away bandwidth and disk space from actually popular files.

I guess clicking once per thunderstorm is ok though.

This post was modified by Ralf Muehlen on 2003-06-20 11:09:41

This post was modified by Ralf Muehlen on 2003-06-20 11:10:08

This post was modified by Ralf Muehlen on 2003-06-20 11:10:37

Reply to this post
Reply [edit]

Poster: raytobe Date: Jun 20, 2003 12:28pm
Forum: freecache Subject: Re: sliding an item into a freecache

How the heck do you open a .tar file? I think I may stick to downloading shows in SHN two tracks at a time,at least if I time out during a download I can pick it up were I left off, seems like I have more success than with this doggone freecache-- anyone?...

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffJonathan Aizen Date: Jun 20, 2003 2:37pm
Forum: freecache Subject: Re: sliding an item into a freecache

WinZip can handle TAR files for Windows. Not sure about the Mac.

Reply to this post
Reply [edit]

Poster: raytobe Date: Jun 20, 2003 2:49pm
Forum: freecache Subject: Re: sliding an item into a freecache

Thanks, finally figured it out and I stand corrected, freecache is muuuuch better than two tracks at a time,saves mucho time.Great program!!

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffbrewster Date: Jun 21, 2003 3:10am
Forum: freecache Subject: .tar and .zip files

these can be opened with winzip on a pc, and stuffit on a mac. if it is not a registered extention for winzip on a pc, this will cause us problems. is everyone having troubles with .tar on a pc?

if so, we should probably change to .zip (but old enough to work with pkzip...)

-brewster

Reply to this post
Reply [edit]

Poster: raytobe Date: Jun 21, 2003 3:14am
Forum: freecache Subject: Re: .tar and .zip files

Seems like I will have to buy the program for windows as the zip decoder that comes with XP will not work with tar

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffJonathan Aizen Date: Jun 21, 2003 3:23am
Forum: freecache Subject: Re: .tar and .zip files

I found two freeware programs that might support tar:

http://qzip.cjb.net/
http://www.coffeecup.com/zip-wizard/

If someone tests and finds out, please post.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or Staffbrewster Date: Jun 21, 2003 6:50am
Forum: freecache Subject: Re: .tar and .zip files


if it is a .tar.gz does winzip recognize it? I dont have a pc here, or I would try it.

-brewster

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffBrak Date: Jun 21, 2003 10:42am
Forum: freecache Subject: Re: .tar and .zip files


PowerArchiver is an alternative that handles .tar .tar.gz and .tar.bz2.

If you are ok with a command line, install CygWin on your Windows box and you can actually use "tar" and "gzip" ;-)

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffDiana Hamilton Date: Jun 21, 2003 10:50pm
Forum: freecache Subject: Re: zip utilities with tar support

Winzip does support .tar, I just installed it yesterday. :)

Previously I used an old version of Netzip which I found does not apparently support .tar.

While browsing around, I found a mention of Easyzip, freeware that supports .tar.

This post was modified by hamilton on 2003-06-22 05:50:11

Reply to this post
Reply [edit]

Poster: lohocla Date: Jun 25, 2003 3:16pm
Forum: freecache Subject: Re: zip utilities with tar support

WinAce handles all these, so does WinZip and WinRar, they are not freeware but have no limitations whatsoever in the nonregistered versions